fix(0228): scale the pivot by the measured USDC/USD rate, pivot-leg reset tooling, XLM snapshot - #311
Conversation
The pivot tier's ref_usd is a price in USDC, not in dollars, so every XLM- and USDT-quoted candle carried the USDC = $1 assumption task 0268 had just removed from the USDC leg itself. USDC closed at 0.9681 on 2023-03-11, so the stored value is ~3.2% high across the whole pre-epoch pivot population. pivot_sql now ASOF-joins prices.usd_rate for canonical USDC at the candidate's bucket END and writes close_usd as close x vwap(ref/USDC) x usdc_usd. The two method-specific rate legs are NESTED, as queries_ch::peg_series_sql proves, so a valid oracle reading wins outright without an argMax across methods. A bucket with neither rate is left unpriced rather than written as 0 x close. The candidate scan moved into a subquery because ASOF needs a materialized column for its inequality, which moves the partition bound onto the bare column and reorders the first two binds.
…d runbook Appendix C The scaled pivot only reaches rows still at close_usd = 0, so the ~118 M pre-epoch XLM-quoted and ~1.6 M USDT-quoted candles already carrying an unscaled value are inert. This adds the operator mechanism that re-opens them. UsdResetSpec gains require_pivot_usdc_rate, which appends the SAME external_rate_day_pred the 0268 mode uses and nothing else: a pivoted row never carried the peg tier's close_usd = close signature, so including it would select nothing. That also means the mode cannot be a fixed point across runs — it is value-idempotent instead, which is a recorded deviation from BRIEF decision C. The mode refuses the canonical USDC leg (that one is 0268's), a non-pivot leg, an oracle-shadowed span, a zero day-set, an empty window, and the two modes together. It is not gated on hourly rates: that refusal guards an irreversibility only the par signature creates. Appendix C carries the preconditions with their STOP conditions, the per-leg --reset-not-before derivation, the dry-run gate, the baseline, the expected runtime and the rollback. post_run_0228_it.rs is the operator's after-check, expected to fail until the campaign runs.
XLM's 52,607 Reflector readings live only in oracle_prices, which the dark
cleanup worker's 13-month policy will eventually prune, while usd_rate — never
pruned — holds none of them. peg_identities() is exactly canonical USDC by
design, and correctly so: XLM is not a peg.
measured_identities() is a second set beside it, named for what its members are:
measured, non-peg references. Its doc comment carries the identity evidence and
a test calls reflector_key_to_identity("XLM") so the claim is code rather than
prose, to task 0267's standard.
The two sets go to populate_usd_rate_from_oracle as SEPARATE calls. Its task
0139 identity pre-pass runs over the whole slice and returns before writing
anything for any identity, so appending XLM to the peg call would let a
collision on XLM's asset_id silently stop USDC's snapshot too.
The copy is gap-filling rather than watermarked, so the first run after deploy
copies all existing readings and no backfill tool is needed.
…ator checklist Implementation notes for the three commits, design decisions (D-01..D-09 from plan, 11 emerged), the D-06 deviation from BRIEF decision C with its evidence, the broken/modified test list, the acceptance-criteria status split between the branch and the operator run, Appendix C's checklist and the spawn list. Records that the #[ignore] ClickHouse tests were written but not run — no ClickHouse is reachable on this machine — and keeps the read-only prod probe that retired RESEARCH assumption A2.
Review WR-01: coarse-repair gated the --pivot-window-s minimum-width guard behind !dry_run, so Appendix C's "dry run first" would accept a window the real run then refuses. The window guards now run in dry-run mode as well; only the snapshot guard stays real-run-only, since a dry run discards nothing. Inherited from the 0182 shape. The task file records the review and verification results.
Two defects found by the prove run against ClickHouse 26.3.10.60. post_run_0228_it decoded a Nullable(Float64) ratio into f64 — the reference vwap's nullIf makes the whole median nullable — so RowBinary landed one byte off and the acceptance gate reported a carried factor of -3.9e230 on a table the tool had just repaired to 0.9681. It could never have passed on prod. The field is Option<f64> now, as in the 0268 twin; NULL behind matched rows is a finding that names the harness, and CI pins it. The ResetRequiresExternalRates refusal both --reset-require-* flags promised was unreachable through coarse-repair: an unloaded series does not fail the month enumeration, it empties it, so the run ended green with "0 month(s)". CoarseRepairDriver::run now checks it first, dry run included, through the one fn the pass already uses.
The #[ignore] ClickHouse suite was recorded as "written, NOT RUN"; it has now run against the prod build, together with a hand-built Appendix C campaign on all six tables and the XLM snapshot path. The task README carries the results, the two defects (Issues 8-9), the FREEZE-after-refusal caveat (Issues 10, spawn list 4) and decisions 21-22. Appendix B and C say what the no-rates refusal now actually does, and C warns about the snapshot a refused run leaves behind.
Three code-review findings, each reproduced before the fix. OracleUsdRatesSnapshotted summed the peg and measured snapshots. XLM lands rows every pass, so a stalled USDC snapshot, the pivot's only post-epoch rate and non-fatal by design, could never read zero. The series is peg-only again and XLM gets OracleMeasuredRatesSnapshotted. The pivot-leg gate checked pivot_ids() alone, while the pivot only runs when canonical USDC resolves in prices.assets. With USDC missing there and its rates loaded, every gate passed and the reset zeroed a leg no pivot refilled (rows_reset 1, rows_enriched 0). It now requires can_pivot(). Both leg refusals ran only inside the per-month pass, which a dry run never builds, so rehearsing the wrong leg ended green. The driver now runs them, with the no-rates check, before enumerating months.
Appendix B and C now list each reset refusal where it actually fires: before connecting, before month enumeration, or inside the pass, which is where the FREEZE-left-behind caveat still applies. The task README records the review (Issues 11), decisions 22-24 and the superseded WR-02 note.
Code review — correctness pass over the diffFour findings. The SQL itself checks out: bind order matches the new render order (candidate watermark → reference watermark → 1.
|
…se in one list Second review round on PR #311, findings 2, 3 and 4. The pivot-leg reset candidate carried no reference-market term, so a bucket on a rate-covered day whose XLM/USDC (or USDT/USDC) market was silent was zeroed and never refilled, caught only by the post-check after the write. The candidate now also requires a usable reference candle on the bucket's UTC day, in the same table, rendered at all three sites like the rate fragment and pinned by its own lockstep test. The dry run still skipped four refusals, including the oracle-shadow guard the runbook calls the campaign's likeliest blocker, because they lived only in reset_step. Every month-independent refusal now lives in assert_reset_is_admissible, run by the driver before enumerating months and by reset_step, so the rehearsal refuses what the real run refuses. The falsifier derived the same baseline variable name for _1m and _1M; the two mechanism grains are matched explicitly. The module header still described the pivot as close x ref_usd; it now carries the USDC/USD factor and the snapshot dependency it introduced.
Second review round on PR #311, finding 1. Since 0228 the enrichment pivot's only post-epoch USDC/USD rate is the oracle worker's usd_rate snapshot of canonical USDC, a copy step that worker treats as non-fatal. A stalled snapshot was invisible from every existing signal: the dark-feed alarm sees rows written, the enrichment backlog alarm sees the oracle tier still enriching, and the pivot tier's own no-progress branch is a warning. One day later every pivot-priced candle lands at close_usd = 0. prices-{env}-oracle-usdc-snapshot-stalled fires when the peg-set snapshot count stays at zero across three hourly buckets while OracleRowsWritten climbs. Hourly because a single pass copying nothing is normal; three hours still leaves most of the pivot's one-day bound. Missing data is not breaching because no passes at all is the dark-feed alarm's case. The snapshot stays non-fatal: failing it would stop the poll and kill both tiers. Deploying this needs the Observability stack, not only EventBridge.
Runbook Appendix B and C: the whole month-independent refusal list runs before enumeration, dry run included; the manual reset-candidate count carries the reference-market term; the abort-signal paragraph names the one residual shape. Task README: decisions 25 to 28, Issues 12, the history entry, the operator checklist now naming the Observability stack, and spawn item 4 demoted since no refusal can fire after a FREEZE any more.
PR #311 merged to develop (d9e25da) and deployed to production on 2026-09-14: EventBridge (nine worker Lambdas) then Observability. The deploy is verified on production: 53,453 XLM readings snapshotted into usd_rate on the first oracle pass, both snapshot metrics published, the stall alarm OK on real data, cleanup still disabled, no Lambda errors, XLM- and USDT-quoted candles priced by the first enrichment pass, and the first coarse-sweep pass in line with the ones before the deploy. Closed with rescoped criterion 2 open: the Appendix C re-enrichment campaign was not run. The spawn list stays a list by decision rather than open tasks, and now carries what the deploy taught for the campaign, including that on 2023-03-11 it moves XLM-quoted candles further from the market because the pivot reference that day is a dust close (0278).
2 participants
Summary
pivot_sqlnow multiplies the ref/USDC vwap by the measured USDC/USD rate fromusd_rateat the bucket end (oracle over external via two nested method-specific ASOF legs, noargMax); a bucket with no rate is left unpriced, neverclose × 1. Fixes the last stored prices still assuming USDC = $1 after 0268 — every pre-epoch XLM- and USDT-quoted candle (2023-03-11 stored +3.2 %).--reset-require-pivot-usdc-rate) for the re-enrichment campaign: one predicate at three sites, five refusals (USDC leg, non-pivot leg, oracle shadow, empty day set, both modes at once), value-idempotent,price_ohlcv_1mstill refused; runbook Appendix C and thepost_run_0228_itfalsifier. Dry run now runs the pivot-window guard too (review WR-01).measured_identities()beside an untouchedpeg_identities()snapshots XLM Reflector readings intousd_rateasoracle/hops 0through a second, separately non-fatal snapshot call; the identity evidence is a test.notes/) rescoped the task: AC 1 as filed is met (pivot vs Reflector ±35 bps), the oracle tier already prices XLM legs inside its window, retention is the dark cleanup-worker's policy, not a TTL.-D warningsclean. The#[ignore]ClickHouse suite HAS run, against a rootlessclickhouse-common-static-26.3.10.60(the prod build):ch_enrich_it44/44,usd_rate_population_it5/5, plus a hand-built Appendix C campaign with the realcoarse-repairbinary on all six tables (0.588 → 0.5692428 = ×0.9681 on 2023-03-11, controls untouched, rerun value-idempotent at version 5) and the XLM snapshot path through the real writer.afe587e,c768b27): thepost_run_0228_itacceptance gate decoded aNullable(Float64)factor intof64and reported ±1e230 on a correctly repaired table — it could never have passed on prod (nowOption<f64>like the 0268 twin, NULL is a harness finding, CI-pinned); and theResetRequiresExternalRatesrefusal both--reset-require-*flags promised was unreachable through the CLI (an unloaded series empties the month enumeration →exit 0, "0 month(s)") —CoarseRepairDriver::runnow checks it before enumerating months, dry run included,#[ignore]-tested and confirmed with the binary. Task README Issues 8–10 and runbook Appendix B/C updated; a refused run still leaves its FREEZE behind locally (Issues 10, spawn list 4, prod unaffected)./code-reviewon the branch: 3 findings, all fixed in9fd74f3/fe452ab, no defect inpivot_sql. Metric: the snapshot counts were summed intoOracleUsdRatesSnapshotted, hiding a stalled USDC snapshot behind XLM's rows — now per set (peg series unchanged in meaning, newOracleMeasuredRatesSnapshotted). Gate: the pivot-leg reset passed with canonical USDC missing fromprices.assetsand zeroed a leg no pivot refills — now requirescan_pivot(). Dry run: both leg refusals (0228 and 0268 modes) ran only in the per-month pass — now before month enumeration.ch_enrich_it46/46 on 26.3.10.60, 725 CI tests.active, a CHORE follows ondevelop. Deploy isPrices-production-EventBridge(enrichment + oracle Lambdas), not Compute.